Statistical method of averaging

[1] 112.2274

Fake Data and the average as an estimate

  • The mean value summarises the path price into a single number
  • But how accurate is this algorithm?
  • The textbook answer is given in terms of the standard error

Standard error function

[1] 0.98525
  • Here averaging is the .content-box-red[algorithm], while the standard error provides the .content-box-blue[inference] of the algorithm’s accuracy.

Explanation

  • It is a surprising, and crucial, aspect of statistical theory that the same data that supplies an estimate can also assess its accuracy.
  • Strictly speaking Inference concerns more than accuracy: recall that algorithms say what the statistician does while inference says why she does it.

Algorithms and Inference

Algorithms and regression

.panelset[ .panel[.panel-name[Least squares algorithm for linear regression] - The least squares estimator is a popular algorithm for estimating a linear regression - The algorithm fits the data by least squares, by minimising the sum of squared deviations over all choices of the model parameters. - Consider the following fake relationship between the price and some market factor

Least squares algorithm

Lowess algorithm for localised regression

  • Lowess is a modern computer based algorithm which works by moving its attention along the x-axis, fitting local polynomial curves of differing degrees to nearby (x,y) coordinates.
  • The fitted estimate above has a similar linear regression as the least squares algorithm in the middle of the data but for higher values of the factor has a much steeper curve.

LS0tCnRpdGxlOiAiQ29tcHV0YXRpb25hbCBub3RlYm9vayBmb3IgUGhEIFNlbWluYXIiCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KYGBge3Igc2V0LXVwfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IEZBTFNFKQpwYWNtYW46OnBfbG9hZCgieGFyaW5nYW5FeHRyYSIsImtuaXRyIiwia2FibGVFeHRyYSIsImZvbnRhd2Vzb21lIiwidGlkeXZlcnNlIiwieGFyaW5nYW50aGVtZXIiKQpgYGAKCgoKIyBTdGF0aXN0aWNhbCBtZXRob2Qgb2YgYXZlcmFnaW5nCgotIENvbnNpZGVyIHRoZSBzaW1wbGlzdGljIGJ1dCBtb3N0IHBvcHVsYXIgbWVhc3VyZSBvZiBhdmVyYWdpbmcgCi0gU3VwcG9zZSB3ZSBvYnNlcnZlIGEgc2V0IG9mIHRoZSBwcmljZSBjaGFuZ2UgcGF0aCBwaGVub21lbm9uIG9mIGEgc3RvY2sgb3ZlciB0IHBlcmlvZHMgJHhfMSwgeF8yLC4uLix4X3QkLgoKYGBge3IgcmFuZG9tLXdhbGstc2ltdWxhdG9yLCBmaWcuaGVpZ2h0PTQsIGZpZy53aWR0aD04fQpzZXQuc2VlZCgxMjMpClMxPTEwMDsgdD0xMDA7IFByaWNlPXZlY3RvcihsZW5ndGggPSB0KQpub2lzZT1ydW5pZih0LG1pbiA9IC01LG1heCA9IDUpCmZvciAodHQgaW4gMTp0KSB7CiAgaWYgKHR0PT0xKSB7CiAgICBQcmljZVt0dF09UzEKICB9CiAgZWxzZXsKICBQcmljZVt0dF0gPSBQcmljZVt0dC0xXSArIG5vaXNlW3R0XQogIH0KfQpzdW0oUHJpY2UpL2xlbmd0aChQcmljZSkKYGBgCgotIFRoZSBhdmVyYWdpbmcgZnVuY3Rpb24gaGlnaGxpZ2h0ZWQgYWJvdmUgaXMgYW4gYWxnb3JpdGhtIGFuZCBpcyByZXByZXNlbnRlZCBieSB0aGUgcmVkIGxpbmUgaW4gdGhlIHBsb3Qgb3Bwb3NpdGUKCiMjIEZha2UgRGF0YSBhbmQgdGhlIGF2ZXJhZ2UgYXMgYW4gZXN0aW1hdGUKCmBgYHtyIGVjaG89RkFMU0UsIGZpZy5oZWlnaHQ9NSwgZmlnLndpZHRoPTd9CnByaWNlX21lYW49c3VtKFByaWNlKS9sZW5ndGgoUHJpY2UpClByaWNlICU+JSBwbG90KHR5cGU9ImwiKSAlPiUgYWJsaW5lKGg9cHJpY2VfbWVhbiwgY29sPSJyZWQiKQpgYGAKCi0gVGhlIG1lYW4gdmFsdWUgc3VtbWFyaXNlcyB0aGUgcGF0aCBwcmljZSBpbnRvIGEgc2luZ2xlIG51bWJlcgotIEJ1dCBob3cgYWNjdXJhdGUgaXMgdGhpcyBhbGdvcml0aG0/Ci0gVGhlIHRleHRib29rIGFuc3dlciBpcyBnaXZlbiBpbiB0ZXJtcyBvZiB0aGUgKnN0YW5kYXJkIGVycm9yKgoKIyMgU3RhbmRhcmQgZXJyb3IgZnVuY3Rpb24KCmBgYHtyLCB3YXJuaW5nPUZBTFNFfQpzdGFuZGFyZF9lcnJvcjwtIGZ1bmN0aW9uKHgpewogIGR4PSh4LW1lYW4oeCkpXjIKICBkZW5vbT1sZW5ndGgoeCkqKGxlbmd0aCh4KS0xKQogIHN1bXNxPXN1bShkeCkvZGVub20KICBzcXJ0KHN1bXNxKQp9CnN0YW5kYXJkX2Vycm9yKFByaWNlKQpgYGAKLSBIZXJlIGF2ZXJhZ2luZyBpcyB0aGUgLmNvbnRlbnQtYm94LXJlZFthbGdvcml0aG1dLCB3aGlsZSB0aGUgc3RhbmRhcmQgZXJyb3IgcHJvdmlkZXMgdGhlIC5jb250ZW50LWJveC1ibHVlW2luZmVyZW5jZV0gb2YgdGhlIGFsZ29yaXRobSdzIGFjY3VyYWN5LgoKIyMgRXhwbGFuYXRpb24KCi0gSXQgaXMgYSBzdXJwcmlzaW5nLCBhbmQgY3J1Y2lhbCwgYXNwZWN0IG9mIHN0YXRpc3RpY2FsIHRoZW9yeSB0aGF0IHRoZSBzYW1lIGRhdGEgdGhhdCBzdXBwbGllcyBhbiBlc3RpbWF0ZSBjYW4gYWxzbyBhc3Nlc3MgaXRzIGFjY3VyYWN5LiAKLSBTdHJpY3RseSBzcGVha2luZyAqSW5mZXJlbmNlKiBjb25jZXJucyBtb3JlIHRoYW4gYWNjdXJhY3k6IHJlY2FsbCB0aGF0IGFsZ29yaXRobXMgc2F5IHdoYXQgdGhlIHN0YXRpc3RpY2lhbiBkb2VzIHdoaWxlIGluZmVyZW5jZSBzYXlzIHdoeSBzaGUgZG9lcyBpdC4KCiMgQWxnb3JpdGhtcyBhbmQgSW5mZXJlbmNlCi0gT2YgY291cnNlLCB0aGUgYHN0YW5kYXJkX2Vycm9yKClgIGZ1bmN0aW9uIGRlZmluZWQgcHJldmlvdXNseSBpcyBpdHNlbGYgYW4gYWxnb3JpdGhtLCB3aGljaCBjb3VsZCBiZSAoYW5kIGlzKSBzdWJqZWN0IHRvIGZ1cnRoZXIgaW5mZXJlbnRpYWwgYW5hbHlzaXMgY29uY2VybmluZyBpdHMgYWNjdXJhY3kgCgotIC5jb250ZW50LWJveC1ncmVlbltUaGUgcG9pbnQgaXMgdGhhdCB0aGUgYWxnb3JpdGhtIGNvbWVzIGZpcnN0IGFuZCB0aGUgaW5mZXJlbmNlIGZvbGxvd3MgYXQgYSBzZWNvbmQgbGV2ZWwgb2Ygc3RhdGlzdGljYWwgY29uc2lkZXJhdGlvbi5dCgotIC5jb250ZW50LWJveC15ZWxsb3dbSW4gcHJhY3RpY2UgdGhpcyBtZWFucyB0aGF0IGFsZ29yaXRobWljIGludmVudGlvbiBpcyBhIG1vcmUgZnJlZS13aGVlbGluZyBhbmQgYWR2ZW50dXJvdXMgZW50ZXJwcmlzZV0KCi0gLmNvbnRlbnQtYm94LWdyZXlbSW4gY29udHJhc3QgaW5mZXJlbmNlIHBsYXlpbmcgY2F0Y2gtdXAgYXMgaXQgc3RyaXZlcyB0byBhc3Nlc3MgdGhlIGFjY3VyYWN5LCBnb29kIG9yIGJhZCwgb2Ygc29tZSBob3QgbmV3IGFsZ29yaXRobWljIG1ldGhvZG9sb2d5Ll0KCgojIEFsZ29yaXRobXMgYW5kIHJlZ3Jlc3Npb24KLnBhbmVsc2V0WwoucGFuZWxbLnBhbmVsLW5hbWVbTGVhc3Qgc3F1YXJlcyBhbGdvcml0aG0gZm9yIGxpbmVhciByZWdyZXNzaW9uXQotIFRoZSBsZWFzdCBzcXVhcmVzIGVzdGltYXRvciBpcyBhIHBvcHVsYXIgYWxnb3JpdGhtIGZvciBlc3RpbWF0aW5nIGEgbGluZWFyIHJlZ3Jlc3Npb24KLSBUaGUgYWxnb3JpdGhtIGZpdHMgdGhlIGRhdGEgYnkgKmxlYXN0IHNxdWFyZXMqLCBieSBtaW5pbWlzaW5nIHRoZSBzdW0gb2Ygc3F1YXJlZCBkZXZpYXRpb25zIG92ZXIgYWxsIGNob2ljZXMgb2YgdGhlIG1vZGVsIHBhcmFtZXRlcnMuCi0gQ29uc2lkZXIgdGhlIGZvbGxvd2luZyBmYWtlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIHRoZSBwcmljZSBhbmQgc29tZSBtYXJrZXQgZmFjdG9yCmBgYHtyIGZha2UgZGF0YX0KZmFjdG9yPC1QcmljZSArIHJ1bmlmKHQsMSw4KV4yICM8PAp0aWJibGUoUHJpY2UsZmFjdG9yKS0+ZGYgCmBgYAoKTGVhc3Qgc3F1YXJlcyBhbGdvcml0aG0KCmBgYHtyIGVjaG89RkFMU0V9CmRmICU+JSBnZ3Bsb3QoYWVzKHk9UHJpY2UseD1mYWN0b3IpKSArIGdlb21fcG9pbnQoKSArCiAgZ2VvbV9zbW9vdGgobWV0aG9kID0gImxtIikgIzw8IGFkZHMgbGVhc3Qgc3F1YXJlcyBsaW5lIHdpdGggc3RhbmRhcmQgZXJyb3JzCmBgYAoKLSBUaGlzIGNvZGUgbWFudWZhY3R1cmVzIGEgcG9zaXRpdmUgcmVsYXRpb25zaGlwIHRvIHRoZSBmYWN0b3IgcGx1cyBzb21lIG5vaXNlIGFuZCB0aGVuIGRyYXdzIHRoZSBsZWFzdCBzcXVhcmVzIHJlZ3Jlc3Npb24gbGluZS4KLSBUaGUgYWNjdXJhY3kgb2YgdGhpcyBlc3RpbWF0ZSBpcyBnaXZlbiBieSAkXHBtJCAyIHN0YW5kYXJkIGVycm9ycwotIFRoZSBhcHByb3ByaWF0ZSBpbmZlcmVuY2Ugb2YgdGhpcyBiYW5kZWQgZ3JleSBhcmVhIGlzIHRoYXQgdGhpcyBoYXMgYSA5NSUgY2hhbmNlIGluIGluY2x1ZGluZyB0aGUgdHJ1ZSBleHBlY3RlZCB2YWx1ZSBvZiBgUHJpY2VgIGluIGFuIGBVcGAgbWFya2V0LgotIFRoaXMgOTUlIGNvdmVyYWdlIGRlcGVuZHMgb24gdGhlIHZhbGlkaXR5IG9mIHRoZSBsaW5lYXIgcmVncmVzc2lvbiBtb2RlbCwgd2hpY2ggY291bGQgYXMgZWFzeSBoYXZlIGJlZW4gYSBxdWFkcmF0aWMgcmVsYXRpb25zaGlwCgojIyBMb3dlc3MgYWxnb3JpdGhtIGZvciBsb2NhbGlzZWQgcmVncmVzc2lvbgoKKiBMb3dlc3MgaXMgYSBtb2Rlcm4gY29tcHV0ZXIgYmFzZWQgYWxnb3JpdGhtIHdoaWNoIHdvcmtzIGJ5IG1vdmluZyBpdHMgYXR0ZW50aW9uIGFsb25nIHRoZSB4LWF4aXMsIGZpdHRpbmcgbG9jYWwgcG9seW5vbWlhbCBjdXJ2ZXMgb2YgZGlmZmVyaW5nIGRlZ3JlZXMgdG8gbmVhcmJ5IGAoeCx5KWAgY29vcmRpbmF0ZXMuIAotIFRoZSBmaXR0ZWQgZXN0aW1hdGUgYWJvdmUgaGFzIGEgc2ltaWxhciBsaW5lYXIgcmVncmVzc2lvbiBhcyB0aGUgbGVhc3Qgc3F1YXJlcyBhbGdvcml0aG0gaW4gdGhlIG1pZGRsZSBvZiB0aGUgZGF0YSBidXQgZm9yIGhpZ2hlciB2YWx1ZXMgb2YgdGhlIGZhY3RvciBoYXMgYSBtdWNoIHN0ZWVwZXIgY3VydmUuCgpgYGB7ciBsb2Vzc30KZGYgJT4lIGdncGxvdChhZXMoeT1QcmljZSx4PWZhY3RvcikpICsgZ2VvbV9wb2ludCgpICsKICBnZW9tX3Ntb290aChtZXRob2QgPSAibG9lc3MiKSAjIApgYGAKCgpgYGB7ciBhbmltYXRpb24gY29kZX0Kc2V0LnNlZWQoMTIzKQpTMT0xMDA7IHQ9MTAwOyBQcmljZT12ZWN0b3IobGVuZ3RoID0gdCkKbm9pc2U9cnVuaWYodCxtaW4gPSAtNSxtYXggPSA1KQpmb3IgKHR0IGluIDE6dCkgewogIGlmICh0dD09MSkgewogICAgUHJpY2VbdHRdPVMxCiAgfQogIGVsc2V7CiAgICBQcmljZVt0dF0gPSBQcmljZVt0dC0xXSArIG5vaXNlW3R0XQogIH0KfQpmYWN0b3I8LVByaWNlICsgcnVuaWYodCwxLDgpXjIgIyA8PAoKbGlicmFyeShhbmltYXRpb24pCmxpYnJhcnkodGlkeXZlcnNlKQphbmkub3B0aW9ucyhubWF4ID0gMjAsIGludGVydmFsID0gMC4wNSkKdGliYmxlKFByaWNlPVByaWNlLGZhY3Rvcj1mYWN0b3IpIC0+IGRmCiAgYm9vdC5sb3dlc3MoeSA9ZGYkUHJpY2UseD1kZiRmYWN0b3IsIGY9Mi8zLGl0ZXIgPSAxMCxsaW5lLmNvbCA9ICJyZWQiLAogICAgICAgICAgICAgIHhsYWIgPSAiZmFjdG9yIiwgeWxhYiA9ICJQcmljZSIpICU+JQogIHNhdmVHSUYobW92aWUubmFtZSA9ICJsb3dlc3NfYW5pLmdpZiIpCgpgYGAKCgo=